Ac 2007-1783: Getting More from Your Data: Application of Item Response Theory to the Statistics Concept Inventory

This paper applies the techniques of Item Response Theory (IRT) to data from the Statistics Concept Inventory (SCI). Based on results from the Fall 2005 post-test (n = 422 students), the analyses of IRT are compared with those of Classical Test Theory (CTT). The concepts are extended to discussions of other applications, such as computerized adaptive testing and Likertscale items which may be of interest to the engineering education community. While techniques based on CTT generally yield valuable information, methods of IRT can reveal unanticipated subtleties in a dataset. For example, items of extreme difficulty (hard or easy) typically attain low discrimination indices (CTT), thus labeling them as “poor”. Application of IRT can identify these items as strongly discriminating among students of extreme ability (high or low). The three simplest IRT models (one-, two-, and three-parameter) are compared to illustrate cases where they differ. The theoretical foundations of IRT are provided, extending to validating the assumptions for the SCI dataset and discussing other potential uses of IRT that are applicable to survey design in engineering education. Introduction The Steering Committee of the National Engineering Education Research Colloquies 1 identified assessment (“Research on, and the development of, assessment methods, instruments, and metrics to inform engineering education practice and learning”) as one of five areas that form the foundation of engineering education research. Further, there are calls for engineering education to become both more inter-disciplinary 2 and rigorous 3 . Item Response Theory 4,5,6,7 (IRT) is commonly used by psychologists in survey design and analysis; effectively “learning the language” opens a conduit for collaboration and dissemination. IRT is therefore useful to place in the engineering education researcher’s toolbelt. The mathematical advantages of IRT enhance rigor with procedures to track characteristics of both the test and examinees across such variables as time, gender, major, etc. This article first describes item response theory from a theoretical perspective, describing the common models for dichotomous (those scored ‘correct’ or ‘incorrect’) items and their assumptions. Secondly, interpretation of IRT is presented by application to the Statistics Concept Inventory (SCI). Finally, some extensions of IRT are described which may be of interest to the engineering educator.